Predictive information, multi-information, and binding information

We introduce an information theoretic measure of dependency between multiple random variables, called ‘binding information’ and compare it with several previously proposed measures of statistical complexity, including excess entropy, Crutchfield et al’s entropy of causal states, Bialek et al’s predictive information, Dubnov’s information rate, and the multiinformation. We discuss and prove some of the properties of the binding information, particularly in relation to the multi-information, and show that, in the case of sets of binary random variables, the processes which maximises binding information are the ‘parity’ processes. Finally we discuss some of the implications this has for the use of the binding information as a measure of complexity. Predictive information, multi-information, and binding information Samer Abdallah ([email protected]) Mark Plumbley ([email protected]) December 9, 2010 Abstract We introduce an information theoretic measure of dependency between multiple random variables, called ‘binding information’ and compare it with several previously proposed measures of statistical complexity, including excess entropy, Crutchfield et al’s entropy of causal states, Bialek et al’s predictive information, Dubnov’s information rate, and the multiinformation. We discuss and prove some of the properties of the binding information, particularly in relation to the multi-information, and show that, in the case of sets of binary random variables, the processes which maximises binding information are the ‘parity’ processes. Finally we discuss some of the implications this has for the use of the binding information as a measure of complexity.We introduce an information theoretic measure of dependency between multiple random variables, called ‘binding information’ and compare it with several previously proposed measures of statistical complexity, including excess entropy, Crutchfield et al’s entropy of causal states, Bialek et al’s predictive information, Dubnov’s information rate, and the multiinformation. We discuss and prove some of the properties of the binding information, particularly in relation to the multi-information, and show that, in the case of sets of binary random variables, the processes which maximises binding information are the ‘parity’ processes. Finally we discuss some of the implications this has for the use of the binding information as a measure of complexity.